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CLAIMS 

What is claimed is: 

1 . A computer-implemented method of reducing the number of hypotheses for 
locations of an active object in a saved image generated by a graphical user interface 

5 (GUI) of an application program comprising: 

determining possible triangles for the saved image, wherein vertices of the 
triangles are points where the active object and any two additional objects of the saved 
image are located; 

determining possible pairs of hypotheses corresponding to the two additional 
10 objects from a current playback image corresponding to the saved image; 

determining triangles corresponding to the possible pairs of hypotheses and 
filtering the detected triangles; 

determining vertices with weights for similar triangles of the current playback 
image; and 

15 calculating a weight for every hypothesis of the active object and filtering 

hypotheses of the active object on the current playback image based on a weight bound. 

2. The method of claim 1, wherein determining possible triangles comprises 
decreasing the number of possible triangles by using only triangles having an edge 
between two additional objects whose length is less than a predetermined distance. 

20 3. The method of claim 1, wherein determining possible pairs of hypotheses 

comprises decreasing the number of possible triangles by using only triangles which 
have an edge between two additional objects whose length is within a calculated 
interval based on a first coefficient value. 

4. The method of claim 1, wherein determining triangles corresponding to the 
25 pairs of hypotheses comprises decreasing the number of triangles by using only 

triangles which have angles within a calculated interval based on a second coefficient 
value. 

5. The method of claim 1, wherein a weight of a selected vertex of a triangle 
comprises 1 .0 multiplied by the number of vertices of other triangles coincident at the 

30 selected vertex. 

6. The method of claim 1, wherein calculating a weight for every hypothesis of 
the active object comprises calculating the sum of distances between the location of the 
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hypothesis and all corresponding vertices using a third coefficient value and an upper 
bound value. 

7. The method of claim 1, wherein filtering the hypotheses of the active object 
comprises filtering a hypothesis if the calculated weight of the hypothesis is less than 

5 the weight bound. 

8. The method of claim 1, further comprising identifying the best hypothesis for 
the active object based on the filtering. 

9. An article comprising: a machine accessible medium containing instructions, 
which when executed, result in reducing the number of hypotheses for locations of an 

10 active object in a saved image generated by a graphical user interface (GUI) of an 
application program by 

determining possible triangles for the saved image, wherein vertices of the 
triangles are points where the active object and any two additional objects of the saved 
image are located; 

1 5 determining possible pairs of hypotheses corresponding to the two additional 

objects from a current playback image corresponding to the saved image; 

determining triangles corresponding to the possible pairs of hypotheses and 
filtering the detected triangles; 

determining vertices with weights for similar triangles of the current playback 
20 image; and 

calculating a weight for every hypothesis of the active object and filtering 
hypotheses of the active object on the current playback image based on a weight bound. 

10. The article of claim 9, wherein instructions to determine possible triangles 
comprise instructions to decrease the number of possible triangles by using only 

25 triangles having an edge between two additional objects whose length is less than a 
predetermined distance. 

1 1 . The article of claim 9, wherein instructions to determine possible pairs of 
hypotheses comprise instructions to decrease the number of possible triangles by using 
only triangles which have an edge between two additional objects whose length is 

30 within a calculated interval based on a first coefficient value. 

12. The article of claim 9, wherein instructions to determine triangles 
corresponding to the pairs of hypotheses comprise instructions to decrease the number 
of triangles by using only triangles which have angles within a calculated interval based 



on a second coefficient value. 

13. The article of claim 9, wherein a weight of a selected vertex of a triangle 
comprises 1.0 multiplied by the number of vertices of other triangles coincident at the 
selected vertex. 

5 14. The article of claim 9, wherein instructions to calculate a weight for every 

hypothesis of the active object comprise instructions to calculate the sum of distances 
between the location of the hypothesis and all corresponding vertices using a third 
coefficient value and an upper bound value. 

15. The article of claim 9, wherein instructions to filter the hypotheses of the 
10 active object comprise instructions to filter a hypothesis if the calculated weight of the 

hypothesis is less than the weight bound. 

16. The article of claim 9, further comprising instructions to identify the best 
hypothesis for the active object based on the filtering. 

17. A cognitive control framework system for automatically controlling 
15 execution of an application program having a graphical user interface comprising: 

a recording component adapted to capture user input data and images displayed 
by the graphical user interface during a recording phase of execution of the application 
program, and to analyze the captured user input data and displayed images to generate 
an execution scenario during the recording phase; and 
20 a playback component adapted to perform image analysis on images displayed 

by the graphical user interface as a result of processing the simulated user input data 
during the playback phase and captured displayed images from the recording phase by, 
the playback component being adapted to reduce the number of hypotheses for 
locations of an active object in a saved image generated by a graphical user interface 
25 (GUI) of an application program by 

determining possible triangles for the saved image, wherein vertices of the 
triangles are points where the active object and any two additional objects of the 
saved image are located; 

determining possible pairs of hypotheses corresponding to the two 
30 additional objects from a current playback image corresponding to the saved 

image; 

determining triangles corresponding to the possible pairs of hypotheses and 
filtering the detected triangles; 
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determining vertices with weights for similar triangles of the current 
playback image; and 

calculating a weight for every hypothesis of the active object and filtering 
hypotheses of the active object on the current playback image based on a weight 
5 bound. 

18. The system of claim 17, wherein determining possible triangles comprises 
decreasing the number of possible triangles by using only triangles having an edge 
between two additional objects whose length is less than a predetermined distance. 

19. The system of claim 17, wherein determining possible pairs of hypotheses 
10 comprises decreasing the number of possible triangles by using only triangles which 

have an edge between two additional objects whose length is within a calculated 
interval based on a first coefficient value. 

20. The system of claim 17, wherein determining triangles corresponding to the 
pairs of hypotheses comprises decreasing the number of triangles by using only 

1 5 triangles which have angles within a calculated interval based on a second coefficient 
value. 

21. The system of claim 17, wherein a weight of a selected vertex of a triangle 
comprises 1 .0 multiplied by the number of vertices of other triangles coincident at the 
selected vertex. 

20 22. The system of claim 17, wherein calculating a weight for every hypothesis 

of the active object comprises calculating the sum of distances between the location of 
the hypothesis and all corresponding vertices using a third coefficient value and an 
upper bound value. 

23. The system of claim 17, wherein filtering the hypotheses of the active object 
25 comprises filtering a hypothesis if the calculated weight of the hypothesis is less than 
the weight bound. 

*24. The system of claim 17, wherein the playback component is adapted to 
identify the best hypothesis for the active object based on the filtering. 



